from sklearn_benchmarks.reporting.hp_match import HpMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HpMatchReporting(against_lib="onnx", config="config.yml", log_scale=True)
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

WARNING! Mismatch between validation scores for 12 predictions (75.0%). See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	1.783428	0.103273	NaN	...	brute	-1	1	0.663	0.308416	0.003663	1.000	5.782541	5.782949	0.337
2	KNeighborsClassifier_brute_force	predict	100000	1	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	0.021181	0.002006	NaN	...	brute	-1	1	1.000	16.883564	0.001345	0.757	0.001255	0.001255	0.243
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	22d5eac4a4146149b35eae6914921514	2cf49829391aa7891e877f2ed070adf0	2.598966	0.019955	NaN	...	brute	-1	5	0.757	16.761322	0.006945	0.882	0.155057	0.155057	0.125
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	1.949392	0.005666	NaN	...	brute	1	100	0.882	0.313454	0.004789	1.000	6.219059	6.219785	0.118
8	KNeighborsClassifier_brute_force	predict	100000	1	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	0.018911	0.000229	NaN	...	brute	1	100	1.000	17.001810	0.027907	0.757	0.001112	0.001112	0.243
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2d5f2b2c02d77766434d9e5033b4a76c	2cf49829391aa7891e877f2ed070adf0	2.601173	0.030045	NaN	...	brute	-1	100	0.882	16.978550	0.001456	0.663	0.153203	0.153203	0.219
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	1.951491	0.003758	NaN	...	brute	1	5	0.757	0.239558	0.004574	1.000	8.146231	8.147715	0.243
14	KNeighborsClassifier_brute_force	predict	100000	1	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	0.019229	0.000361	NaN	...	brute	1	5	1.000	3.598588	0.025056	0.922	0.005343	0.005344	0.078
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	b79d63bb7670c492de5c3befac58fe29	2cf49829391aa7891e877f2ed070adf0	1.075387	0.001489	NaN	...	brute	1	1	0.663	3.637775	0.012878	0.929	0.295617	0.295619	0.266
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	1.590791	0.025237	NaN	...	brute	-1	1	0.896	0.239020	0.004227	1.000	6.655466	6.656506	0.104
20	KNeighborsClassifier_brute_force	predict	100000	1	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	0.004262	0.001410	NaN	...	brute	-1	1	1.000	3.554953	0.029420	0.922	0.001199	0.001199	0.078
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	22d5eac4a4146149b35eae6914921514	88843f54689e3271092f70126e1de585	2.381358	0.080133	NaN	...	brute	-1	5	0.922	3.635965	0.009732	0.896	0.654945	0.654948	0.026

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.0	7.374	0.0	-1	1	16.947	0.132	0.663	0.001	0.001	See	See
3	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.0	7.396	0.0	-1	5	0.312	0.006	1.000	0.035	0.035	See	See
6	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.0	7.447	0.0	1	100	17.231	0.091	0.882	0.001	0.001	See	See
9	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.0	7.413	0.0	-1	100	0.299	0.010	1.000	0.036	0.036	See	See
12	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.0	7.268	0.0	1	5	3.639	0.006	0.896	0.003	0.003	See	See
15	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.0	7.441	0.0	1	1	0.240	0.004	1.000	0.045	0.045	See	See
18	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.004	0.0	0.380	0.0	-1	1	3.639	0.008	0.929	0.001	0.001	See	See
21	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.004	0.0	0.370	0.0	-1	5	0.240	0.004	1.000	0.018	0.018	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.783	0.103	0.000	0.002	-1	1	0.308	0.004	1.000	5.783	5.783	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.021	0.002	0.000	0.021	-1	1	16.884	0.001	0.757	0.001	0.001	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.599	0.020	0.000	0.003	-1	5	16.761	0.007	0.882	0.155	0.155	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.024	0.004	0.000	0.024	-1	5	0.309	0.005	1.000	0.076	0.076	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.949	0.006	0.000	0.002	1	100	0.313	0.005	1.000	6.219	6.220	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.019	0.000	0.000	0.019	1	100	17.002	0.028	0.757	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.601	0.030	0.000	0.003	-1	100	16.979	0.001	0.663	0.153	0.153	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.022	0.002	0.000	0.022	-1	100	0.313	0.004	1.000	0.071	0.071	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.951	0.004	0.000	0.002	1	5	0.240	0.005	1.000	8.146	8.148	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.019	0.000	0.000	0.019	1	5	3.599	0.025	0.922	0.005	0.005	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.075	0.001	0.001	0.001	1	1	3.638	0.013	0.929	0.296	0.296	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.018	0.000	0.000	0.018	1	1	0.240	0.005	1.000	0.074	0.074	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.591	0.025	0.000	0.002	-1	1	0.239	0.004	1.000	6.655	6.657	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.004	0.001	0.000	0.004	-1	1	3.555	0.029	0.922	0.001	0.001	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.381	0.080	0.000	0.002	-1	5	3.636	0.010	0.896	0.655	0.655	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.007	0.005	0.000	0.007	-1	5	0.239	0.004	1.000	0.028	0.028	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

WARNING! Mismatch between validation scores for 12 predictions (75.0%). See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.750415	1.013010	NaN	...	kd_tree	-1	1	0.929	2.618640	0.209573	1.000	0.286567	0.287483	0.071
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.002514	0.000358	NaN	...	kd_tree	-1	1	1.000	117.608244	0.000000	0.946	0.000021	0.000021	0.054
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	dc74b969bf622bc24ba3bc62c980983b	1eb6cf1a720a225efb91df0b529b0510	0.980333	0.344856	NaN	...	kd_tree	-1	5	0.946	118.961067	0.000000	0.951	0.008241	0.008241	0.005
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	4.806605	0.401040	NaN	...	kd_tree	1	100	0.951	2.625837	0.165090	1.000	1.830504	1.834118	0.049
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	0.003213	0.000850	NaN	...	kd_tree	1	100	1.000	123.634710	0.000000	0.946	0.000026	0.000026	0.054
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	022f7445d43bb1dbc24dc3106c03cb93	1eb6cf1a720a225efb91df0b529b0510	2.817059	0.201310	NaN	...	kd_tree	-1	100	0.951	119.091454	0.000000	0.929	0.023655	0.023655	0.022
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	1.502982	0.268727	NaN	...	kd_tree	1	5	0.946	0.005021	0.000085	1.000	299.353830	299.397009	0.054
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	0.001462	0.000320	NaN	...	kd_tree	1	5	1.000	0.037454	0.000618	0.911	0.039022	0.039027	0.089
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	a9455565db1d8e052a783317c99744ff	1eb6cf1a720a225efb91df0b529b0510	0.831806	0.334570	NaN	...	kd_tree	1	1	0.929	0.060016	0.002260	0.894	13.859842	13.869663	0.035
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.026699	0.015512	NaN	...	kd_tree	-1	1	0.891	0.005073	0.000089	1.000	5.263054	5.263864	0.109
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.001908	0.000125	NaN	...	kd_tree	-1	1	1.000	0.037855	0.000152	0.911	0.050415	0.050415	0.089
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	dc74b969bf622bc24ba3bc62c980983b	edf7cb014c2c0910e1c3aaaf8ae633f2	0.022506	0.000939	NaN	...	kd_tree	-1	5	0.911	0.037336	0.000293	0.891	0.602784	0.602803	0.020

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	2.743	0.043	0.029	0.0	-1	1	118.790	0.000	0.929	0.023	0.023	See	See
3	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.287	0.057	0.024	0.0	-1	5	2.626	0.234	1.000	1.252	1.257	See	See
6	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.263	0.035	0.025	0.0	1	100	119.915	0.000	0.951	0.027	0.027	See	See
9	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.354	0.077	0.024	0.0	-1	100	2.628	0.167	1.000	1.276	1.279	See	See
12	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.337	0.035	0.024	0.0	1	5	0.038	0.000	0.891	87.860	87.862	See	See
15	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.247	0.046	0.025	0.0	1	1	0.005	0.000	1.000	643.038	643.133	See	See
18	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.002	0.016	0.0	-1	1	0.059	0.000	0.894	0.017	0.017	See	See
21	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.000	0.031	0.0	-1	5	0.005	0.000	1.000	0.101	0.101	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.750	1.013	0.000	0.001	-1	1	2.619	0.210	1.000	0.287	0.287	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.000	0.000	0.003	-1	1	117.608	0.000	0.946	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.980	0.345	0.000	0.001	-1	5	118.961	0.000	0.951	0.008	0.008	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	5	2.608	0.147	1.000	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	4.807	0.401	0.000	0.005	1	100	2.626	0.165	1.000	1.831	1.834	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	1	100	123.635	0.000	0.946	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	2.817	0.201	0.000	0.003	-1	100	119.091	0.000	0.929	0.024	0.024	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.001	0.000	0.005	-1	100	2.638	0.221	1.000	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.503	0.269	0.000	0.002	1	5	0.005	0.000	1.000	299.354	299.397	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	5	0.037	0.001	0.911	0.039	0.039	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.832	0.335	0.000	0.001	1	1	0.060	0.002	0.894	13.860	13.870	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	1	0.005	0.000	1.000	0.189	0.189	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.027	0.016	0.001	0.000	-1	1	0.005	0.000	1.000	5.263	5.264	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	1	0.038	0.000	0.911	0.050	0.050	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.023	0.001	0.001	0.000	-1	5	0.037	0.000	0.891	0.603	0.603	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	5	0.005	0.000	1.000	0.400	0.400	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

WARNING! Mismatch between validation scores for 1 prediction (100.0%). See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	max_leaf_nodes	min_samples_leaf	n_iter_no_change	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	25b9f14bed7dd5d830c6ccd4dfebbf0c	8c8fffa8ec4b2d8de833421f9e32beab	0.104758	0.000344	300	...	100	100	10	0.824	0.3936	0.009678	1.0	0.266154	0.266234	0.176

1 rows × 25 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	fit	100000	100000	100	91.702	0.0	300	0.001	0.001	0.476	0.023	0.824	192.793	193.013	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.105	0.0	300	0.008	0.0	0.394	0.01	1.0	0.266	0.266	See	See